When user goals are known, but the flows and screens required to get there are still forming, going to external testing seems a disproportionate burden on resources, and relying only on individual internal opinions is risky. Cognitive walkthroughs give a reliable, proven structure to the review discussions which teams often already have as part of their inherent process, giving project outcomes greater validity and certainty.
Although the cognitive walkthrough method is tailored to internal participants, the essential concepts can also be adapted for larger, external group testing.
If you’re working in or learning about digital product development, especially if your project is new, complex, or requires interaction by a very broad range of users, then cognitive walkthroughs are here to help you reach the optimal product more efficiently.
Why you should conduct a Cognitive Walkthrough
Cognitive walkthroughs are a great way to bring some rigor to your discussions and evaluations of how well a new product or feature will help users. As a new product is being developed, to some extent, everyone in the team is concerned with creating the optimum way for users to achieve the goals which the product enables.
Countless meetings can be spent discussing options and setting the direction for how a product will serve users – a process which is heavily influenced by personalities, tastes, and opinions. While that may lead to some great results, it lacks the validity of a more formal, consistent, and hopefully objective process, which is exactly what cognitive walkthroughs bring.
Using cognitive walkthroughs as your early-stage testing method should give your project a more thoughtfully built foundation, while at the same time being quick and very low-cost to undertake. Your cognitive walkthroughs should hopefully uncover issues which would be increasingly costly to address as the project evolves.
Furthermore, as a method which uses rough prototypes, and often participants from within your own org, cognitive walkthroughs are a very efficient way to sense-check novel ideas and complex flows.
Many UX testing methods can be used at any stage throughout a product’s development and lifetime. However, as a product or feature matures, it naturally takes on visual style and layout, which means that later-stage testing will be influenced by UI presentation and not just focused on essential interactions.
Cognitive walkthrough is well-suited to very rough wireframe prototypes, because in a classic walkthrough, the testing group is internal, and therefore may be more forgiving of ‘unprofessional’ prototypes. Also, cognitive walkthroughs are conducted in a small group with a presenter, who can quickly explain things which may be missing from the prototype.
Cognitive walkthroughs take the participants out of their day-to-day working routine so that they can take a step back and evaluate the product they are working on. In this more reflective state of mind, new ideas can be uncovered, as well as aspects of the product which need improvement may be found, since they are often harder to see when working closely on project delivery tasks.
What is Cognitive Walkthrough
A cognitive walkthrough is a usability evaluation method used to examine the ease of learning a product by exploring its functionality through the lens of a first-time user. This method focuses specifically on understanding the cognitive processes that users engage in when interacting with an interface, particularly when they are attempting to use it without prior training. The goal is to identify potential usability issues that could hinder a new user’s ability to effectively use the system.
How Cognitive Walkthrough Works:
- Define User Tasks: The evaluation begins with clearly defining the tasks that users are expected to perform with the product. These tasks should be specific and representative of typical user interactions.
- Develop User Action Sequences: For each task, the evaluator outlines a logical sequence of steps that a user would need to take to complete the task successfully. This sequence forms the basis of the walkthrough.
- Perform the Walkthrough: During the walkthrough, the evaluator (often a usability expert) simulates a user’s thought process at each step of the task. The focus is on four key questions:
- Will the user try to achieve the right effect? Does the user know what to do at this step?
- Will the user notice that the correct action is available? Is it clear what actions can be taken next?
- Will the user associate the correct action with the effect that the user is trying to achieve? Does the interface communicate how actions lead to outcomes?
- If the correct action is performed, will the user see that progress is being made toward the solution of the task? Does the system provide feedback that helps the user understand that they are on the right track?
- Identify Usability Problems: At each step, the evaluator identifies where a user might encounter difficulties due to the design of the interface. These issues could be related to the visibility of controls, the labeling and grouping of information, the feedback provided by the system, or the assumptions made about the user’s knowledge.
- Recommend Improvements: Based on the findings, recommendations are made to improve the user interface, enhance user understanding and interaction, and ensure that
Watch our video to learn more about the cognitive walkthrough and how to conduct it:
How are Cognitive Walkthroughs different from other usability tests?
At a glance, cognitive walkthroughs may appear similar to a prototype user test, or other usability tests. In some settings, they may, in fact, be very similar, but if you’re following the documented methods, there are a number of differences.
Cognitive Walkthroughs | Other Usability Tests |
Focused on learnability | Can assess any aspect of the UX |
Audience is new users | Can be for new or inducted users |
Participants are internal, with set roles | External testing pool, reflecting real users |
Small testing group | May use the largest group possible |
Suited to low-fi mockups | Often offers a working prototype |
Heuristic Testing compared to Cognitive Walkthrough
Heuristic Evaluation (Nielsen and Molich, 1990; Nielsen 1994) is a method created to test how well an interface performs relative to a set of defined principals, which are accepted to be the basis of good user experience.
Cognitive walkthroughs are aimed at evaluating new features, systems, and approaches, and also have a different structure. Heuristic Evaluation can provide feedback on how easy your product is to use, while cognitive walkthroughs are a formal review of how easy it is to learn.
What can be evaluated with a Cognitive Walkthrough?
Cognitive walkthroughs provide insights to any aspect of a product which requires users to perform a new task. They are suitable at a whole-product, feature, or single-screen level.
Typically, the technique is used when:
- The product, feature, or screen requires users to learn how to use it
- The UX addresses a problem in a way that is different to existing norms
- The user is required to complete complex workflows, or
- It is likely that the user is unfamiliar or completely new to this kind of flow
When to use Cognitive Walkthroughs in product development
As the outcomes can be high-impact and the prototypes can be rudimentary, cognitive walkthrough is well-matched to conceptual work phases. Furthermore, as a cognitive walkthrough is so easy to set up and administer, the technique can be used throughout product design and development, even on deployed products at a relatively high frequency.
Just be aware that the outcomes are shaped by the testing group, which is classically a limited, internal set.
Cognitive Walkthrough participant group options
In principle, a Cognitive Walkthrough is done by a small group of people who work in the product organization. This may be designers, product managers, engineers, IAs, department leads, in fact, anyone who may have a relevant perspective. Sticking with this group should make walkthroughs simple to organize and true to their original form and benefits.
However, it is also possible to conduct a cognitive walkthrough with larger groups, as well as with testers from outside the product organization. As long as the cognitive walkthrough participant, question, and method structure are maintained, you should still gain excellent qualitative and quantitative results.
This approach takes longer to organize and conduct, and is unlikely to be conducted quickly or often. Moreover, it’s important to source a pool of relevant users, which in the case of cognitive walkthrough means that they are new to the media being tested and are in the product target audience.
Where did Cognitive Walkthrough come from?
For as long as people have been inventing new systems, they have tested how efficient those systems are for new users. Early software engineers created processes to review and test the sequence of actions that users need to take in order to interact with their projects, which is the origin of cognitive walkthrough for product development.
In the 1990s, a group of researchers created a walkthrough methodology initially in reference to assessing ‘walk-up-and-use’ interfaces, such as ATMs. During the 90s, they evolved the method to be directly orientated to websites and coined the term ‘Cognitive Walkthrough’. Today, that same format is used in the evaluation of all kinds of digital products, with different practitioners adding their own refinements and variants.
The Cognitive Walkthrough method
Cognitive walkthrough is a formal method: it has a defined structure for whoever participates, what their roles are, how the session runs, and what questions are to be asked and discussed. The method is tried and tested and should yield meaningful results and confidence.
However, as your experience with cognitive walkthrough grows, you can alter aspects of the format to suit your workflow and situation. Be careful, though, to maintain the integrity of the general structure to help ensure the quality of outcomes.
The participants in a Cognitive Walkthrough
In its original form, a cognitive walkthrough comprises participants who represent the broader product team.
This includes:
- Product experts who know all about this product
- Domain experts who know all about this market
- One or more UX practitioners
- One or more Engineers
In the modern context, there is also room for product managers, product owners, UI designers, UX writers, information architects, UX researchers, and anyone else on a product squad. Certain organizations may also involve senior managers, founders, and even those in sales and marketing.
And, in actual practice, the participants are often only a group of product or UX designers!
The four questions always used in Cognitive Walkthroughs
The fundamental characteristic of a cognitive walkthrough is the questions which are to be discussed. Their original wording is still useful and relevant; however, there is a simplified and more universal version from Chris Kimmer which we prefer.
You can use either, or you may find further options for all or any of the four questions that suit you better.
Cognitive Walkthrough original question | …in simple terms | |
1 | Will users be trying to produce whatever effect the action has? | Will the user be trying to achieve the right effect? |
2 | Will users see the control (button, menu, switch, etc.) for the action? | Is the action visible? |
3 | Once users find the control, will they recognize that it produces the effect they want? | Will the user recognize the action as the correct one? |
4 | After the action is taken, will users understand the feedback they receive so that they can go on to the next action with confidence? | Will the user understand the feedback? |
Cognitive Walkthrough planning
A small amount of planning for your cognitive walkthrough will help the session run well and maximize your chances of meaningful results. A good cognitive walkthrough agenda will cover the following information.
1. Overview
- Purpose – The purpose of the workshop, what is being evaluated, and why?
- Audience – Who are the personas that the evaluators should empathize with?
- Requirements – List the prototypes, tools and spaces to be used
- Rigor – Are there any challenges to research rigor and objectivity, and how will they be overcome?
- Outcomes – Outline what artifacts will be created and their use.
2. People
List the specific people required to do the workshop, and what role each will play. It is acceptable for the same person to perform multiple roles. A cognitive walkthrough requires the following roles to be performed.
- Facilitator – Runs the session.
- Presenter – Operates the prototype.
- Recorder – Takes notes and collates them after the session.
- Evaluators – Discuss the four questions.
3. Subjects
Make a list or table of each sequence or goal to be evaluated. Note if there are any aspects specific to each sequence, such as a different persona, scenario, or prototype.
Preparing a cognitive walkthrough prototype
Although we’re only touching on it lightly here, preparing the prototype or prototypes for a walkthrough workshop should be the main preparation activity. It may not be a painstaking task though; remember, these prototypes only need to be as finished as is necessary to demonstrate the concept.
Once you have completed the above agenda and your prototype, you’re ready to conduct the workshop. Typically, the agenda is circulated either before or at the beginning of the session, however, be mindful of the potential of this kind of information to influence the results. You should always strive to create an agenda, but if you feel it will skew the workshop outcomes, there is no need to circulate it.
How to conduct a Cognitive Walkthrough workshop
The final aspect of the cognitive walkthrough method is the workshop itself. The following sequence is recommended for a classic ‘internal’ workshop.
- The Facilitator starts by talking through the goals, roles, personas, rules, and optionally, the agenda.
- The Presenter steps through the interface, stopping at each decision point.
- The Evaluators discuss each of the four questions and agree to an action at each step.
- The Presenter takes the agreed actions.
- The Recorder notes success or failure, and any qualitative feedback.
- The Recorder summarizes and circulates the outcomes after the workshop.
Ideally, your workshop will run closely to this format, but in practice, you should expect it to vary a lot from the format. The outcomes are still valid, and more so if you’ve been able to keep to the four questions.
Workshops with external participants will need to be adapted from this format to suit the specific setting.
Tips for conducting a Cognitive Walkthrough workshop:
- Overemphasize personas to the evaluators – it’s important that they’re empathetic with your actual new users.
- Keep laptops closed – we need full attention!
- Do not try to justify the prototype – participants should talk about what they encounter, not why it is the way it is, with strong focus on user reaction.
- When testing with internal participants it can be helpful to really limit the number of participants.
- Conversely, a larger group of external participants can create more meaningful results.
- Test microcopy options, not just button placement.
- For an external walkthrough, consider using a mixed-method approach, such as a survey or interview to add depth to observations.
Conclusion: The benefits of Cognitive Walkthroughs in collaborative work
Applying a cognitive walkthrough structure to the normal practice of stepping back from your work and discussing it with colleagues adds diligence and improved potential to impact quality at a comparatively low cost.
From their genesis in the early-days of software programming to their fascinating development for walk-up-and-use systems, as well as their current use in assessing digital products, walkthroughs are a proven, valuable, and enduring tool.
In order to streamline your research and empathize with users better, register for your free UXtweak account and give a try to our various testing tools!
Sources:
[Clayton Lewis, Peter Polson, Cathleen Wharton, John Reiman, Testing a Walkthrough Methodology for Theory-Based Design of Walk-Up-and-Use Interfaces, 1990 https://www.cs.helsinki.fi/u/thusu/opinnot/kaytarv/artikkelit/p235-lewis.pdf ]
[Clayton Lewis, Peter Polson, Cathleen Wharton, John Reiman, The cognitive walkthrough: A practitioner’s guide, 1994 https://www.colorado.edu/ics/sites/default/files/attached-files/93-07.pdf ]
